Image processing method and device

ABSTRACT

An image processing device acquires at least two first images and down-samples the at least two first images to obtain at least two second images, where a first resolution of the at least two first images is higher than a second resolution of the at least two second images. By using the at least two first images and the at least two second images, the image processing device respectively determines a first depth map corresponding to the at least two first images under a limit of a first disparity threshold, and a second depth map corresponding to the at least two second images under a limit of a second disparity threshold, where the second disparity threshold is greater than the first disparity threshold. The image processing device then combines the determined first depth map with the second depth map to generate a combined depth map.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present disclosure is a continuation of U.S. application Ser. No.16/822,937, filed on Mar. 18, 2020, which is a continuation ofInternational Application No. PCT/CN2017/103630, filed Sep. 27, 2017,the entire contents of both of which are incorporated herein byreference.

TECHNICAL FIELD

The present disclosure relates to the field of image processingtechnology and, more particularly, to a method and device for imageprocessing.

BACKGROUND

With the development of computer technology, as an important field ofintelligent computing, computer vision has been greatly developed andapplied. Computer vision relies on imaging systems instead of visualorgans as input sensitive means. Among these imaging systems, camerasare the most commonly used ones. For example, a dual vision camera maybe used to form a basic vision system.

Currently, a corresponding depth map may be generated by using abinocular camera system through two images taken by two cameras at twodifferent angles at the same time.

In the actual process of calculating a depth map, the depth map isusually calculated within a certain search region to reduce thecalculation. However, for high-resolution images, this process causesnearby objects to be unrecognizable. If the search region is broadened,the amount of calculation will be extremely large. For low-resolutionimages, limiting the search region will result in a low observationaccuracy, especially for observation of the distant objects.

SUMMARY

In accordance with the present disclosure, there is provided an imageprocessing device. The image processing device includes a memory and aprocessor. The processor is configured to acquire at least two firstimages, where a resolution of the at least two first images is a firstresolution. The processor also acquires at least two second images,where a resolution of the at least two second images is a secondresolution, where the second resolution is lower than the firstresolution. By using the at least two first images, the processordetermines a first depth map corresponding to the at least two firstimages under a limit of a first disparity threshold. By using the atleast two second images, the processor further determines a second depthmap corresponding to the at least two second images under a limit of asecond disparity threshold, where the second disparity threshold isgreater than the first disparity threshold. The determined first depthmap and the second depth map are then combined by the processor togenerate a combined depth map.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a schematic diagram of a method for depth calculationaccording to an embodiment of the present disclosure;

FIG. 2 illustrates a flowchart of a method for image processingaccording to an embodiment of the present disclosure;

FIG. 3 illustrates a low-resolution image and a corresponding depth mapaccording to an embodiment of the present disclosure;

FIG. 4 illustrates a high-resolution image and a corresponding depth mapaccording to an embodiment of the present disclosure;

FIG. 5 illustrates a combined depth map according to an embodiment ofthe present disclosure;

FIG. 6 illustrates a schematic diagram of a position for ato-be-processed region in an image according to an embodiment of thepresent disclosure;

FIG. 7 illustrates a schematic diagram of a position for ato-be-processed region in an image according to another embodiment ofthe present disclosure;

FIG. 8 illustrates a schematic diagram of a position for ato-be-processed region in an image according to yet another embodimentof the present disclosure;

FIG. 9 illustrates a schematic diagram of a position for ato-be-processed region in an image according to yet another embodimentof the present disclosure;

FIG. 10 illustrates a schematic diagram of an image block segmentationprocessing of a high-resolution image according to an embodiment of thepresent disclosure;

FIG. 11 illustrates a schematic diagram of an image block segmentationprocessing of a high-resolution image according to another embodiment ofthe present disclosure;

FIG. 12 illustrates a schematic block diagram of an image processingdevice according to an embodiment of the present disclosure;

FIG. 13 illustrates a schematic block diagram of an image processingdevice according to another embodiment of the present disclosure; and

FIG. 14 illustrates a schematic block diagram of an unmanned aerialvehicle according to an embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

To make the objective, technical solutions, and advantages of thepresent disclosure clearer, the technical solutions of the embodimentsof the present disclosure will be made in detail hereinafter withreference to the accompanying drawings of the disclosed embodiments.Apparently, the disclosed embodiments are merely some, but not all, ofthe embodiments of the present disclosure. Various other embodimentsobtained by a person of ordinary skills in the art based on theembodiments of the present disclosure without creative efforts stillfall within the protection scope of the present disclosure.

Unless otherwise stated, all technical and scientific terms used in theexamples of the present disclosure have the same meanings as commonlyunderstood by those skilled in the relevant art of the presentdisclosure. The terms used in the present disclosure are merely for thepurpose of describing specific embodiments, and are not intended tolimit the scope of the present disclosure.

Computer vision relies on imaging systems instead of visual organs asinput sensitive means. Among these imaging systems, cameras are the mostcommonly used ones. For example, a dual vision camera may be used toform a basic vision system.

A corresponding depth map may be generated by taking pictures fromdifferent angles at the same time using two cameras of a binocularcamera system. The binocular camera system may be a front-view binocularcamera system, a rear-view binocular camera system, a left-viewbinocular camera system, or a right-view binocular camera system.

In the actual process of calculating a depth map, a matching calculationmay be performed based on two images taken by two cameras at the sametime, and the depth information of each pixel in the images iscalculated.

Optionally, a depth of a pixel may be calculated by using the followingEquation (1):

$\begin{matrix}{d = {f\frac{b}{d_{p}}}} & (1)\end{matrix}$

where d is the depth, b is the distance between the left and rightcameras, f is the focal length of the cameras, and d_(p) is thedisparity.

As can be seen from the above Equation (1), since b and f are physicalproperties and generally remain unchanged, d is inversely proportionalto d_(p). For a nearby object, the depth is smaller and the disparity islarger, while for a distant object, the depth is larger while thecorresponding disparity is smaller.

An example regarding how to calculate a depth will be describedhereinafter with reference to FIG. 1.

As shown in FIG. 1, a pixel, in the right image, that matches a pixel inthe left image is to be located. That is, search and traverse on astraight line in the right image to find a pixel that matches the pixelin the left image, i.e., a pixel with the highest match score values inthe right image. A disparity between the pixel in the left image and thematching pixel in the right image is then calculated.

It is to be understood that FIG. 1 only shows a single match. In theactual process, the pixels in an image may be searched one by one. Inaddition, in FIG. 1, only a local matching is conducted. In the actualprocess, after the matching, optimization and adjustment may be furtherperformed, to eventually calculate a disparity for a pixel between theleft and right images.

For example, as shown in FIG. 1, a pixel on the nasal tip of a mask islocated at row 20, column 100 in the left image. After the left andright images are rectified, theoretically, the pixel on the nasal tip inthe right image should be located also on row 20 but the column positionshould be <100. Accordingly, by searching from right to left startingfrom the pixel at row 20, column 100, a pixel at row 20, column 80 inthe right image that matches the pixel on the nasal tip in the left magemay be eventually determined. The disparity of the determined pixel is|80-100|=20.

As can be seen from the above, for a high-resolution image, it takes along time to perform matching calculations if each pixel on each row ofthe image is calculated. Therefore, in the actual calculations, a searchregion may be limited. For instance, a search is limited to a maximum of64 disparities on an image with a resolution of 320*240. That is, eachpixel in the left image just needs to be searched 64 times in the rightimage. Accordingly, by limiting the maximum search region, thecalculation time required for the matching calculations will be reduced,thereby lowering the consumption of the computing resources.

However, for a high-resolution image, for example, for an image with aresolution of 640*480, if a search is still limited to a maximum of 64disparities, it will cause nearby objects to be unrecognizable, that is,a large dead zone will appear. If the search region is broadened, therequired amount of calculation will be quite large.

For a low-resolution image, for example, for an image with a resolutionof 320*240, limiting a search to a maximum of 64 disparities will resultin lower observation accuracy for distant objects. This can be seen fromEquation (1). For a distant object, that is, an object with a smalldisparity, e.g., a disparity of only 2, a disparity error of ±0.5 willmake the calculated depth greatly deviate from the actual depth.However, for a nearby object, e.g., an object with a disparity of 30,a±0.5 disparity error will not make the calculated depth greatly deviatefrom the actual depth.

From the above analysis, it can be seen that, for a nearby object, ifthe search is limited to a maximum of 64 disparities for an image with aresolution of 320*240, then for an image with a resolution of 640*480,the search needs to be limited to a maximum of 128 disparities. Thiswill lead to a skyrocketing of the required computing resources. For adistant object, for an image with a resolution of 640*480, if the searchis limited to a maximum of 2 disparities, then for an image with aresolution of 320*240, the search needs to be limited to a maximum of 1disparity, which then results in a really low observation accuracy.

To observe nearby objects more accurately and to observe distant objectswith a higher observation accuracy, for images with a resolution of640*480, the search needs to be limited to a maximum of 128 disparities,which requires a large amount of calculation. For an aircraft that has ahigh demand for real-time processing, this is quite challenging toachieve.

When an aircraft flies at a low altitude, the aircraft needs to avoidobstacles that are within a short distance. Meanwhile, a depth mapcalculated by using high-resolution images may not be helpful due to thelarge dead zones. On the other hand, when the aircraft is flying at ahigh speed, a high accuracy is required for the observation of distantobjects. At this moment, a depth map calculated using low-resolutionimages cannot meet this requirement. Under certain circumstances,low-resolution images may be used to calculate a depth map, but thisrequires an aircraft to limit its flight speed.

For the above reasons, the embodiments of the present disclosure providean image processing solution, which acquires more accurate depthinformation by combining depth maps generated from high- andlow-resolution images, and does not require a large amount ofcalculation.

FIG. 2 is a flowchart of an image processing method 100 according to anembodiment of the present disclosure. The method 100 includes at least apart of the following description.

Step 110: Acquire at least two first images, where a resolution of theat least two first images is a first resolution.

Optionally, the at least two first images may originate from a binocularcamera. For example, the at least two first images may be images takenby a binocular camera at the same time, or may be images down-sampledfrom the images taken by the binocular camera at the same time.

It is to be understood that the at least two first images may notnecessarily originate from a binocular camera. For example, the at leasttwo first images may originate from a monocular or a multiocular (morethan binocular) camera.

Step 120: Acquire at least two second images, where a resolution of theat least two second images is a second resolution, and the secondresolution is lower than the first resolution.

Optionally, the at least two second images may be acquired bydownsampling the at least two first images, respectively.

Optionally, the at least two first images and the at least two secondimages may be respectively generated by downsampling images with ahigher resolution.

Step 130: Use the at least two first images to determine a first depthmap corresponding to the at least two first images under the limit of afirst disparity threshold.

Specifically, the first disparity threshold may be considered as amaximum search region. On a first image, a pixel matching a certainpixel in another first image is searched to find the disparitycorresponding to that pixel, so as to get the depth for that pixel.

The value of the depth or depth information described in the embodimentsof the present disclosure may be a depth d or a disparity in Equation(1). This is because the disparity has an inverse relationship with thedepth d, and that the disparity directly reflects the depth.

Specifically, a depth map described in the embodiments of the presentdisclosure may directly include the depth d of each pixel or include thedisparity corresponding to each pixel.

Step 140: Use the at least two second images to determine a second depthmap corresponding to the at least two second images under a limit of asecond disparity threshold, where the second disparity threshold isgreater than the first disparity threshold.

Specifically, the second disparity threshold may be used as a maximumsearch region. On a second image, a pixel matching a certain pixel ofanother second image is searched, so as to find a disparitycorresponding to that pixel, so as to get the depth for that pixel.

Step 150: Combine the first depth map and the second depth map togenerate a combined depth map.

Optionally, the combination of the first depth map and the second depthmap may use the following approach:

Use the depths of a first portion of pixels on the first depth map andthe depths of a second portion of pixels on the second depth map togenerate a combined depth map. Here, the first portion of pixels arepixels, on the first depth map, that match a third portion of pixels,where the third portion of pixels are the pixels other than the secondportion of pixels on the second depth map.

Specifically, in the above approach, the depth information of oneportion of pixels on the second depth map and the depth information ofcertain pixels, on the first depth map, that match the other portion ofpixels on the second map may be used to generate a combined depth map.

It is to be understood that the combination of depth maps in theembodiments of the present disclosure is not limited to the abovedescribed approach. For instance, the depth information of a certainpixel on the first depth map and the depth information of a pixel on thesecond depth map that matches the certain pixel on the first map may becombined and processed (i.e., two depth information are combined, forexample, through averaging or weighted processing, etc.) to acquire thedepth information for that pixel.

Optionally, the disparities corresponding to the depths of the thirdportion of pixels described above are less than or equal to a thirddisparity threshold.

Specifically, because the second depth map calculated by using alow-resolution image and under the limit of a larger disparity thresholdis less accurate for distant objects or people (i.e., theircorresponding disparities are smaller), the depth information for thedistant part may be replaced with the depth information of the matchedpixels in the first depth map, so that the problem of low accuracy forthe depth information for the distant part may be solved.

Optionally, the third disparity threshold is equal to a value obtainedby dividing the first disparity threshold by a first value. Here, thefirst value is a pixel ratio of the first resolution to the secondresolution in a first direction, where the first direction is a pixelscanning direction when acquiring the first depth map and the seconddepth map.

Optionally, if a depth map is acquired by scanning in rows, the firstdirection is a row direction. If a depth map is acquired by scanning incolumns, the first direction is a column direction. Apparently, thescanning direction may also be other directions, which are notspecifically limited in the embodiments of the present disclosure.

For example, if the resolution of the first images is 640*480, and theresolution of the second image sis 320*240, and the depth map is scannedin rows, then the first value may be 2.

Optionally, in the embodiments of the present disclosure, the depths ofthe second portion of pixels may be maintained on the second depth map.On the second depth map, the depths corresponding to values obtained bydividing the disparities corresponding to the depths of the firstportion of pixels by the first value may be used to replace the depthsof the third portion of pixels.

It is to be understood that, in addition to a value obtained by dividingthe first disparity threshold value by the first value, the value forthe third disparity threshold may also be other values, for example, avalue smaller than that of the first disparity threshold divided by thefirst value.

It is to be understood that, in the foregoing descriptions, the depthsof the third portion of pixels are replaced on the basis of the seconddepth map. However, under certain circumstances, the embodiments of thepresent disclosure may not necessarily change the depth information ofsome pixels on the basis of the second depth map, but rather re-recordthe depth information of the first portion of pixels and the depthinformation of the second portion of pixels on a new depth map.

To better understand the present disclosure, a method for calculating adepth map in the present disclosure will be described hereinafter byusing the first images with a resolution of 640*480 and a firstdisparity threshold of 8 disparities and the second images with aresolution of 320*240 and a second disparity threshold of 64 disparitiesas an example.

Step 1: Calculate a depth map from at least two images with a lowresolution. That is, a depth map is generated based on the images with aresolution of 320*240 and under a limit of 64 disparities.

Specifically, after the original images with a resolution of 640*480 aredown-sampled to images with a resolution of 320*240, a depth map is thengenerated under a limit of 64 disparities.

For example, the left part in FIG. 3 is an image with a resolution of320*240 (one of the at least two images), and the right part of FIG. 3is the corresponding depth map calculated by the applicant. From thedepth map in FIG. 3, it can be seen that the nearby ground is relativelysmooth, but the distant ground has a clear stair-like structure. Thatis, the accuracy of the depth information calculated for the distantpart is not really high.

Step 2: Use high-resolution images, but under a stricter limit ofdisparity threshold. That is, make a depth map based on images with aresolution of 640*480 and under a limit of 8 disparities. The purposehere is to calculate the points for the distant part.

For example, the left part in FIG. 4 is an image with a resolution of640*480 (i.e., one of the at least two images). The right part of FIG. 4is the corresponding depth map calculated by the applicant. A searchwith only 8 disparities is made on the high-resolution images. From thedepth map shown in the right part of FIG. 4, it can be seen thatalthough the nearby ground has some flaws, the distant ground is quitesmooth.

Step 3: Combine the depth map generated from the high-resolution imagesand the depth map generated from the low-resolution images. That is, onthe depth map generated from the low-resolution images, replace thedisparities or depths for the points of less than 4 disparities with thedisparities or depths of the corresponding points on the depth mapgenerated from the high-resolution images.

That is, on the depth map generated from the low-resolution images, thepoints with a depth corresponding to a disparity greater than 4 areretained in the original calculation, but the depths of the points witha depth corresponding to a disparity less than or equal to 4 arereplaced with the depths obtained by dividing the disparities of thematched pixels on the depth map corresponding to the high-resolutionimages by 2.

For example, FIG. 5 illustrates a depth map generated after the depthmaps in FIG. 3 and FIG. 4 are combined. The result here is close to aresult obtained by the applicant by directly calculating the depths ofthe high-resolution images using 128 disparities.

It is to be understood that the darker the gray color in FIG. 3 to FIG.5, the greater the depth. However, because only the grayscale diagram isused for illustration, the color shade in some places may not have agood correlation with the corresponding depths.

Optionally, in some embodiments of the present disclosure, whenprocessing an image, for certain reasons (e.g., the processingcapability and processing efficiency of the system), the image needs tobe segmented or a to-be-processed region needs to be intercepted fromthe image. The segmented image block(s) or the intercepted region isthen used to calculate the depth map.

To facilitate understanding, the following two implementations will bedescribed in detail in combination. However, it is to be understood thatthe process of image segmentation or intercepting a to-be-processedregion is not limited to these two implementations illustrated in theembodiments of the present disclosure. In addition, certain features ofthe following two implementations may be used in combination unlessthere are some clear conflicts.

Implementation I

Perform a segmentation processing on each first image of at least twofirst images to obtain segmented image blocks; combine image blocks witha same position on the at least two first images to obtain a pluralityof image block groups; determine the depth map of each image block groupin the plurality of image block groups under a limit of the firstdisparity threshold; join the depth maps of the plurality of image blockgroups together to obtain the first depth map.

Optionally, each first image may be segmented respectively according tothe processing capability of the system (e.g., the maximum computingcapability of the computing unit in the system).

Specifically, because the maximum computing capacity of the computingunit of the system is limited, if the resolution of an image is high andthe size of the image is large, the calculation of the depth map may bevery difficult to perform. Accordingly, a high-resolution image may besegmented, to allow each segmented image block to meet the maximumcomputing capacity of the computing unit.

Optionally, the image segmentation described in the embodiments of thepresent disclosure may be a uniform image segmentation. Apparently, thesegmentation may not be necessarily always uniform. In one example, thesegmentation is performed sequentially according to the maximumcomputing capacity of the computing unit until the last remaining imageblock that requires a computing capacity less than or equal to themaximum computing capacity of the computing unit.

Optionally, a plurality of computing units may perform a parallelprocessing on the obtained plurality of image block groups to acquiredepth information corresponding to each image block group, therebyimproving the image processing efficiency.

It is to be understood that although the above embodiment has beendescribed in conjunction with the segmentation of the first images as anexample, the embodiments of the present disclosure are not limitedthereto. Although the resolution of the second images is lower than thatof the first images, the second images may also be subjected to thesegmentation processing (e.g., the computing capacity required for thesecond images is still greater than the maximum computing capacity ofthe computing unit in the system), and the segmented image block groupsfor the second images are used to calculate the depth map. The specificprocess for the second images may be similar to those described abovefor the first images. In other words, the second images may also besubjected to the segmentation processing.

Implementation II

On each of the at least two first images, a to-be-processed region isdetermined respectively; and the to-be-processed regions from the atleast two first images are used to determine the first depth map under alimit of the first disparity threshold.

Optionally, the to-be-processed regions are determined according to theprocessing capability of the system.

Specifically, because the maximum computing capacity of the computingunit of the system is limited, if the resolution of an image is high andthe size of the image is large, it will be very difficult to performdepth calculations. Accordingly, a to-be-processed region is obtainedfrom each image according to the maximum computing capacity of thecomputing unit of the system.

Optionally, an expected moving position of a movable object isestimated, and the to-be-processed regions in the first images aredetermined according to the expected moving position of the movableobject.

Optionally, the movable object may be an aircraft, an auto-driving car,or the like.

Optionally, the at least two first images are obtained by photo-shootingby a photographing device mounted on the movable object. The currentspeed of a reference object in the photographing device coordinatesystem is obtained and used to estimate the expected moving position ofthe movable object.

Optionally, the current speed of the movable object is used to estimatethe current speed of the reference object in the photographing devicecoordinate system.

For example, the current moving speed of the movable object may beobtained through an inertial measurement unit installed on the movableobject, so as to estimate the current speed of the reference object inthe photographing device coordinate system.

Optionally, the current speed of the reference object in thephotographing device coordinate system is estimated by using the movingtrajectory of the movable object.

For example, the previously moved positions of the movable object may beobtained first. In the next, the points of the moved positions areprojected into the photographing device coordinate system. The speed ofthe reference object in the photographing device coordinate system isthen calculated based on the position change of the points in a seriesof captured image frames.

Optionally, the reference object may be a reference object that isstationary with respect to the earth, or a reference object that ismoving with respect to the earth. Optionally, the reference object maybe an obstacle that needs to be avoided by the moveable object.

Optionally, according to the speed of the reference object at time A inthe photographing device coordinate system, a position P of the center Gof the reference object in the photographing device coordinate system attime B (time B is after time A) may be estimated. The position P isprojected to an image captured by the photographing device at time A,and is recorded as p. A to-be-processed region centered around p andhaving a specified region size is then determined.

Specifically, an expected moving position in the image may be estimatedaccording to the speed of the reference object in the photographingdevice coordinate system. Since [v_(x) ^(c), v_(y) ^(c), v_(z) ^(c)] isknown, and the focal length f of the camera is also known, according toa similar triangle relationship, Equation (2) may be:

$\begin{matrix}\left\{ \begin{matrix}{{\Delta u} = {f\frac{v_{x}^{c}}{v_{z}^{c}}}} \\{{\Delta v} = {f\frac{v_{y}^{c}}{v_{z}^{c}}}}\end{matrix} \right. & (2)\end{matrix}$

With the offset [Δu, Δv]^(T), and based on the optical axis coordinate[u₀, v₀]^(T) (the original center point) of the first image given by thecalibration parameters, the center [u₀+Δu, v₀+Δv]^(T) of theto-be-processed region may be calculated. Next, according to thespecified region size, by using [u₀+Δu, v₀+Δv]^(T) as the center point,an image with the specified region size is then intercepted. For moredetails, refer to FIG. 6 and FIG. 7.

In one implementation, if the region matching the expected movingposition exceeds a first image, the region matching the expected movingposition is modified to obtain a to-be-processed region having thespecified region size on the first image. For example, a to-be-processedregion is shown in FIG. 8. In the figure, the black-filled region is theto-be-processed region, and the larger rectangular frame is the regionof the first image.

In another implementation, if the region matching the expected movingposition exceeds a first image, a sub-region, that does not exceed thefirst image, within the region matching the expected moving position isdetermined as the to-be-processed region. For example, a to-be-processedregion is shown in FIG. 9. In the figure, the black-filled region is theto-be-processed region, and the larger rectangular frame is the regionof the first image.

Optionally, the specified region size is determined according to theprocessing capability of the system. For example, the specified regionsize is equal to the maximum computing capacity of the computing unit ofthe system.

It is to be understood that although the foregoing embodiments aredescribed by taking a to-be-processed region selected from a first imageas an example, the embodiments of the present disclosure are not limitedthereto. Although the resolution of the second images is lower than thatof the first images, a to-be-processed region may also be selected froma second image (e.g., the computing capacity required by a second imageis still greater than the maximum computing capacity of the computingunit), and the depth map is calculated based on the to-be-processedregion on the second image. The specific process may be similar to theabove description with respect to the first images. In other words, thesecond images may also be intercepted.

Optionally, in the embodiments of the present disclosure, an image groupmay be selected from a plurality of image groups according to a movingdirection of the movable object, where the selected image group includesat least two first images.

Specifically, the movable object may have a plurality of photographingsystems, and images that need to perform depth information combinationmay be selected according to the moving direction of the movable object.

For example, assuming that the movable object needs to move forward, agroup of images captured by a front-view camera may be selected. Theselected group of images may be used to generate depth mapscorresponding to the high- and low-resolution images, and then the depthinformation of the corresponding depth maps may be combined.

For example, assuming that the movable object needs to move in the frontleft direction, a group of images taken by a front-view camera and agroup of images taken by a left-view camera may be then selected. Depthmaps corresponding to the respective high- and low-resolution images arerespectively generated by using the two groups of images. Accordingly,the depth maps are respectively generated for the two groups of images,which are then respectively combined.

Optionally, the depth maps in the embodiments of the present disclosuremay be used to avoid obstacles.

Optionally, the combined depth map in the embodiments of the presentdisclosure may be combined with another non-combined depth map to avoidobstacles.

Specifically, at least two third images are acquired, and the thirdimages have the second resolution (i.e., the low resolution). Use the atleast two third images to determine a third depth map corresponding tothe third images under a limit of the second disparity threshold. Thethird depth map and the combined depth map are used to avoid obstacles.

The third images may not be in the moving direction of the movableobject, for example, a direction opposite to the moving direction of themovable object.

For example, assuming that the movable object needs to move forward, agroup of images taken by a front-view camera may be selected. Theselected group of images are used to generate depth maps correspondingto the high- and low-resolution images. The depth information on thedepth maps are then combined to avoid the obstacles ahead. A group ofimages taken by a rear-view camera may also be selected, and thelow-resolution images are used to generate a depth map under a limit ofa large disparity threshold, to avoid obstacles in the back.

For example, assuming that the movable object needs to move in the frontleft direction, a group of images taken by a front-view camera and agroup of images taken by a left-view camera may be selected, and used togenerate depth maps corresponding to the respective high- andlow-resolution images of the two groups of images. The depth mapsrespectively generated from the two groups of images are then combined,to avoid obstacles in the front left direction of movement. Meanwhile, agroup of images taken by a rear-view camera are selected, and thelow-resolution images and a large disparity threshold limit are used togenerate a depth map, to avoid obstacles in the back. A group of imagestaken by a right-view camera are selected, and low-resolution images anda large disparity threshold limit are used to generate a depth map, toavoid obstacles on the right.

To facilitate understanding, the following description will be madebased on two specific embodiments in combination with an aircraft in aspecific scenario. It is to be understood that the two specificembodiments described below are only for the convenience of the readerto understand the present disclosure, and should not be constructed aslimiting the present disclosure.

Background information for the following two embodiments: Originalimages obtained by the sensor(s) during the actual process arehigh-resolution images, that is, a resolution of 1280*800 (WXGA, or800p). To ensure that the depth information is able to be used ascontrol feedback, it may be optimal to have a certain calculationfrequency of the depth map (e.g., 10 Hz (i.e., 10 frames per second,frame interval 100 ms)). However, due to the limitation of the computingresources on an aircraft, the computing unit supports images with amaximum resolution of 640*480 (VGA). In addition, a maximum of 6 groupsof images may be calculated in 100 ms.

Embodiment 1 (Avoid Front and Rear Obstacles)

Step 1: First, down-sample two groups of high-resolution WXGA images,taken by the front-view and rear-view cameras, to VGA images to obtaintwo groups of low-resolution images.

Step 2: According to the direction of flight, select a front-view imagegroup (when flying forward) or a rear-view image group (when flyingbackward), and segment each WXGA image included in the selected imagegroups into four pieces, each of which is slightly smaller than a VGAimage. Accordingly, four images are obtained for each WXGA image. Here,each WXGA image is segmented but not down-sampled. Actually, it may beconsidered that each WXGA image is divided into 4 calculations for 4depth maps, which are then joined together to form a depth map for theWXGA image. Therefore, this step is equivalent to calculating depth mapsfor high-resolution images, and thus a stricter limit of disparitythreshold should be selected. Among the segmented images, image blocksin the same position may form an image block group. For example, asshown in FIG. 10 and FIG. 11, WXGA1 and WXGA2 are segmented,respectively. Image block 1-1 and image block 2-1 form an image blockgroup, image block 1-2 and image block 2-2 for an image block group,image block 1-3 and image block 2-3 form an image block group, and imageblock 1-4 and image block 2-4 form an image block group.

Step 3: The two image groups in Step 1 and the four image block groupsin Step 2 (exactly six image groups or the image block groups in total)are each calculated for its respective depth map by the computing unit.Next, a depth map calculated from a VGA image downsampled from ahigh-resolution WXGA image in Step 1 is used as the basis map, which isthen combined with the depth map calculated from the group of four smallsegmented image blocks to get a more accurate depth map.

Embodiment 2 (Avoid Obstacles in all Directions)

Step 1: Down-sample the four groups of high-resolution WXGA images takenfrom the front, back, left, and right sides to VGA images, to get 4groups of low-resolution images.

Step 2: According to the direction of flight, select a first image groupof front-view images (taken from a forward, front left, or front rightflight) or rear-view images (taken from a backward, rear left, or rearright flight), and select a second group of left-view images (taken forma left, front left, or rear left flight) or right-view images (takenfrom a right, front right or rear right), to get two groups ofhigh-resolution WXGA images. Next, predict the flight trajectory of themovable object based on the flight speed. According to the flighttrajectory or direction, select a VGA image for each image of the firstimage group to form a first image block group, and select a VGA imagefor each image of the second image group to form a second image blockgroup.

Step 3. For the four image groups selected in Step 1 and the two imageblock groups in Step 2, calculate their respective depth maps throughthe computing unit. Take the two depth maps calculated based on VGAimages down-sampled from the two groups of high-resolution WXGA images(the direction of the view of the maps selected here is the same as thedirection of the view of the maps in Step 2) as the basis maps, andcombine them (may be combined in each direction) with the respectivedepth maps generated from the two image block groups in Step 2, so as toget more accurate depth maps.

In the embodiments of the present disclosure, for high-resolutionimages, a smaller disparity threshold is used for the depth mapcalculations, and for low-resolution images, a larger disparitythreshold is used for the depth map calculations. The depth mapgenerated based on the high-resolution images and the depth mapgenerated based on the low-resolution images are combined. Accordingly,the problem of a large dead zone of a depth map calculated based on ahigh image resolution and a small disparity threshold (the selection ofsuch calculation simply for the reason to save the calculation) may besolved by the depth information calculated based on a low imageresolution and a large disparity threshold. Meanwhile, the problem oflow accuracy of depth information for a distant part calculated with alow image resolution and a large disparity threshold may be solved bythe depth information calculated based on a high image resolution and asmall disparity threshold. Accordingly, the image processing method ofthe embodiments of the present disclosure acquires more accurate depthinformation by combining the depth maps generated from high- andlow-resolution images, which does not require a large amount ofcalculation (e.g., due to the use of depth maps based on low-resolutionimages), and can also solve the problem of necessary flight speed limitin order to avoid obstacles by an aircraft.

FIG. 12 is a schematic block diagram of an image processing adviceaccording to an embodiment of the present disclosure. As shown in FIG.12, the device includes an image acquisition unit 310, a depthcalculation unit 320, and a depth combination unit 330.

The image acquisition unit 310 is configured to: acquire at least twofirst images, where a resolution of the first images is a firstresolution; and acquire at least two second images, where a resolutionof the second images is a second resolution, and the second resolutionis lower than the first resolution.

The depth calculation unit 320 is configured to: use the at least twofirst images to determine a first depth map corresponding to the atleast two first images under a limit of a first disparity threshold; anduse the at least two second images to determine a second depth mapcorresponding to the at least two second images under a limit of asecond disparity threshold, where the second disparity threshold isgreater than the first disparity threshold.

The depth combination unit 330 is configured to combine the first depthmap and the second depth map to generate a combined depth map.

Optionally, the deep combination unit 330 is further configured to:combine the depths of a first portion of pixels on the first depth mapand the depths of a second portion of pixels on the second depth map togenerate the combined depth map. Here, the first portion of pixels arethe pixels, on the first depth map, that match a third portion ofpixels, where the third portion of pixels are the pixels other than thesecond portion of pixels on the second depth map.

Optionally, disparities corresponding to the depths of the third portionof pixels are less than or equal to a third disparity threshold.

Optionally, the third disparity threshold is equal to a value obtainedby dividing the first disparity threshold by a first value, where thefirst value is a pixel ratio of the first resolution to the secondresolution in a first direction, where the first direction is a pixelscanning direction when the first depth map and the second depth map areacquired.

Optionally, the deep combination unit 330 is further configured to:maintain the depths of the second portion of pixels on the second depthmap; and replace the depths of the third portion of pixels on the seconddepth map with depths corresponding to values obtained by dividing thedisparities corresponding to the depths of the first portion of pixelsby the first value.

Optionally, the depth calculation unit 320 is further configured to:perform a segmentation processing in each of the at least two firstimages to obtain segmented image blocks; combining at least two imageblocks having a same position on the first images to obtain a pluralityimage block groups; determine a depth map of each image block group inthe plurality of image block groups under a limit of the first disparitythreshold; and join the depth maps of the plurality of image blockgroups together to generate the first depth map.

Optionally, the depth calculation unit 320 is further configured tosegment each of the first images respectively according to theprocessing capability of the system.

Optionally, the depth calculation unit 320 is further configured to:determine a to-be-processed region on each of the at least two firstimages, respectively; and use the to-be-processed regions of the atleast two first images to determine the first depth map under a limit ofthe first disparity threshold.

Optionally, the depth calculation unit 320 is further configured todetermine a to-be-processed region according to the processing capacityof the system.

Optionally, the depth calculation unit 320 is further configured to:estimate the expected moving position of a movable object; and determinethe to-be-processed region on the first images according to the expectedmoving position.

Optionally, the depth calculation unit 320 is further configured to:take the expected moving position as the center and determining a regionmatching the expected moving position on the first image according to aspecified region size; and when the region matching the expected movingposition exceeds the first image, the region matching the expectedmoving position is modified to obtain a to-be-processed region havingthe specified region size on the first image.

Optionally, the depth calculation unit 320 is further configured to:take the expected moving position as the center and determine a regionmatching the expected moving position on the first image according tothe specified region size; and when the region matching the expectedmoving position exceeds the first image, a sub-region, within the regionmatching the expected moving position, that does not exceed the firstimage is determined as the to-be-processed region.

Optionally, the depth calculation unit 320 is further configured todetermine the specified region size according to the processing capacityof the system.

Optionally, the at least two first images are obtained by aphotographing device on the movable object; and the depth calculationunit 320 is further configured to: obtain the current speed of areference object in the photographing device coordinate system; estimatethe expected moving position according to the current position of thereference object in the photographing device coordinate system.

Optionally, the depth calculation unit 320 is further configured to: usethe current moving speed of the movable object to estimate the currentspeed of the reference object in the photographing device coordinatesystem; or, use the already moved positions of the movable object toestimate the current speed of the reference object in the photographingdevice coordinate system.

Optionally, the processing capacity of the system is the maximumcomputing capacity of the computing unit of the system.

Optionally, the image acquisition unit 310 is further configured to:acquire at least two third images, where the third images have a secondresolution; and the depth calculation unit 320 is further configured touse the at least two third images to determine a third depth mapcorresponding to the third images under a limit of the second disparitythreshold. As shown in FIG. 12, the image processing device furtherincludes an obstacle avoiding unit 340 that is configured to use thethird depth map and the combined depth map to avoid obstacles.

Optionally, the image acquisition unit 310 is further configured todown-sample the at least two first images to obtain the at least twosecond images.

Optionally, the image acquisition unit 310 is further configured toselect an image group from a plurality of image groups according to amoving direction of the movable object, where the selected image groupincludes the at least two first images.

It is to be understood that the image processing device 300 may executethe solutions and steps described in the method 100. For brevity, therelated details are not described herein again.

FIG. 13 is a schematic block diagram of an image processing device 400according to another embodiment of the present disclosure.

Optionally, the image processing device 400 may include a plurality ofdifferent components, which may be integrated circuits (ICs), or partsof integrated circuits, discrete electronic devices, or other circuitboard (such as motherboard or add-on board)-compatible modules that mayserver as an integrated part of a computer system.

Optionally, the image processing device may include a processor 410 anda storage medium 420 coupled to the processor 410.

The processor 410 may include one or more general-purpose processors,such as a central processing unit (CPU), or a processing device.Specifically, the processor 410 may be a complex instruction setcomputing (CISC) microprocessor, a very long instruction word (VLIW)microprocessor, and a microprocessor for implementing a plurality ofinstruction set combinations. The processor may also be one or morespecial-purpose processors, such as application specific integratedcircuits (ASICs), field programmable gate arrays (FPGAs), digital signalprocessors (DSPs).

The processor 410 may communicate with the storage medium 420. Thestorage medium 420 may be a magnetic disk, an optical disk, a read onlymemory (ROM), a flash memory, or a phase change memory. The storagemedium 420 may store instructions stored by the processor, and/or maycache some information stored from an external storage device, forexample, layered pixel information of an image of a pyramid read from anexternal storage device.

Optionally, in addition to the processor 410 and the storage medium 420,the image processing device may further include a displaycontroller/display unit 430, a transceiver 440, a video input/outputunit 450, an audio input/output unit 460, and other input/output units470. These components included in the image processing device 400 may beinterconnected through a bus or an internal connection.

Optionally, the transceiver 440 may be a wired transceiver or a wirelesstransceiver, such as a WIFI transceiver, a satellite transceiver, aBluetooth transceiver, a wireless cellular phone transceiver, or acombination thereof.

Optionally, the video input/output unit 450 may include an imageprocessing subsystem, such as a camera. The image processing subsystemmay include a light sensor, a charged coupled device (CCD), or acomplementary metal-oxide semiconductor (CMOS) light sensor forphoto-shooting functions.

Optionally, the audio input/output unit 460 may include a speaker, amicrophone, a headphone, and the like.

Optionally, the other input/output devices 470 may include a storagedevice, a universal serial bus (USB) port, a serial port, a parallelport, a printer, a network interface, and the like.

Optionally, the image processing device 400 may perform the operationsshown in the method 100. For brevity, the related details are notdescribed herein again.

Optionally, the image processing device 300 or 400 may be located on amoving device. The moving device may move in any suitable environment,for example, in the air (e.g., a fixed-wing aircraft, a rotorcraft, oran aircraft with neither a fixed-wing nor a rotor), in the water (e.g.,a ship or a submarine), on land (e.g., car or train), space (e.g., spaceplane, satellite, or space probe), and any combination of the aboveenvironments. The moving device may be an aircraft, such as an unmannedaerial vehicle (UAV). In some embodiments, the moving device may carry alive subject, such as a human or an animal.

FIG. 14 is a schematic block diagram of a moving device 500 according toan embodiment of the present disclosure. As shown in FIG. 14, the movingdevice 500 includes a carrier 510 and a load 520. The description of themoving device as a UAV in FIG. 14 is for illustrative purposes only. Theload 520 may be connected to the moving device without being through thecarrier 510. The movable device 500 may further include a propulsionsystem 530, a sensing system 540, a communication system 550, an imageprocessing device 562, and a photographing system 564.

The propulsion system 530 may include an electronic speed controller(may also be referred to as ESC), one or more propellers, and one ormore electric motors coupled to the one or more propellers. The motorsand the propellers are disposed on the corresponding arms. The ESC isconfigured to receive a driving signal generated by a flight controllerand provide a driving current to the motors according to the drivingsignal, to control the rotation speed and/or steering of the motors. Themotors are configured to drive the propellers to rotate, so as toprovide propulsion for the UAV flight. The propulsion allows the UAV toachieve one or more degrees of freedom of movement. In some embodiments,the UAV may be rotated around one or more rotational axes. For example,the rotational axes may include a roll axis, a yaw axis, and a pitchaxis. It is to be understood that a motor may be a DC motor or an ACmotor. In addition, a motor may be a brushless motor or a brushed motor.

The sensing system 540 is configured to measure the attitude informationof the UAV, that is, the position information and status information ofthe UAV in space, such as three-dimensional position, three-dimensionalangle, three-dimensional velocity, three-dimensional acceleration, andthree-dimensional angular velocity. The sensing system may includesensors, for example, at least one of a gyroscope, an electroniccompass, an inertial measurement unit (“IMU”), a vision sensor, a globalpositioning system (“GPS”), and a barometer. The flight controller isconfigured to control the UAV flight. For example, the UAV flight may becontrolled according to the attitude information measured by the sensingsystem. It is to be understood that the flight controller may controlthe UAV according to a pre-programmed program instruction, and may alsocontrol the UAV by responding to one or more control instructions fromthe control device.

The communication system 550 may communicate with a terminal device 580having a communication system 570 through a wireless signal 590. Thecommunication system 550 and the communication system 570 may include aplurality of transmitters, receivers, and/or transceivers for wirelesscommunication. The wireless communication here may be a one-waycommunication. For example, only the moving device 500 may send data tothe terminal device 580. Alternatively, the wireless communication mayalso be a two-way communication, through which the data may be sent fromthe moving device 500 to the terminal device 580, or from the terminaldevice 580 to the moving device 500.

Optionally, the terminal device 580 may provide control data for one ormore of the moving device 500, the carrier 510, and the load 520, andmay receive information sent by the moving device 500, the carrier 510,and the load 520. The control data provided by the terminal device 580may be used to control the state of the one or more of the moving device500, the carrier 510, and the load 520. Optionally, the carrier 510 andthe load 520 include a communication module for communicating with theterminal device 580.

It is to be understood that the image processing device 660 included inthe moving device shown in FIG. 14 may execute the method 100. Forbrevity, the related details are not described herein again.

The foregoing descriptions are merely specific implementations of thepresent disclosure, but the protection scope of the present disclosureis not limited thereto. Any person skilled in the art may easily deriveother variations or substitutions within the technical scope disclosedin the present disclosure, all of which shall fall within the protectionscope of the present disclosure. Accordingly, the protection scope ofthe present disclosure shall be subject to the protection scope of theappended claims.

What is claimed is:
 1. An image processing device, comprising: a memoryfor storing program instructions; and a processor coupled to the memoryto recall the program instructions that, when executed by the processor,cause the processor to perform a method including: acquiring at leasttwo first images, wherein a resolution of the at least two first imagesis a first resolution; down-sampling the at least two first images toobtain at least two second images, wherein a resolution of the at leasttwo second images is a second resolution, and the second resolution islower than the first resolution; using the at least two first images todetermine a first depth map corresponding to the at least two firstimages under a limit of a first disparity threshold; using the at leasttwo second images to determine a second depth map corresponding to theat least two second images under a limit of a second disparitythreshold, wherein the second disparity threshold is greater than thefirst disparity threshold; and combining the first depth map with thesecond depth map to generate a combined depth map.
 2. The imageprocessing device according to claim 1, wherein combining the firstdepth map with the second depth map to generate the combined depth mapfurther includes: using depths of a first portion of pixels on the firstdepth map and depths of a second portion of pixels on the second depthmap to generate the combined depth map, wherein the first portion ofpixels are pixels on the first depth map that match a third portion ofpixels, and the third portion of pixels are pixels other than the secondportion of pixels on the second depth map.
 3. The image processingdevice according to claim 2, wherein disparities corresponding to depthsof the third portion of pixels are less than or equal to a thirddisparity threshold.
 4. The image processing device according to claim3, wherein: the third disparity threshold is equal to a value obtainedby dividing the first disparity threshold by a first value; the firstvalue is a pixel ratio of the first resolution to the second resolutionin a first direction; and the first direction is a pixel scanningdirection when the first depth map and the second depth map areacquired.
 5. The image processing device according to claim 4, whereinusing the depths of the first portion of pixels on the first depth mapand the depths of the second portion of pixels on the second depth mapto generate the combined depth map further includes: maintaining thedepths of the second portion of pixels on the second depth map; and onthe second depth map, replacing the depths of the third portion ofpixels with depths corresponding to values obtained by dividingdisparities corresponding to the depths of the first portion of pixelsby the first value.
 6. The image processing device according to claim 1,wherein using the at least two first images to determine the first depthmap corresponding to the at least two first images under the limit ofthe first disparity threshold further includes: performing asegmentation processing in each of the at least two first images toobtain segmented image blocks; grouping image blocks having samepositions in the at least two first images to obtain a plurality ofimage block groups; determining a depth map of each image block group inthe plurality of image block groups under the limit of the firstdisparity threshold; and joining depth maps of the plurality of imageblock groups together to generate the first depth map.
 7. The imageprocessing device according to claim 6, wherein performing thesegmentation processing in each of the at least two first images furtherincludes: performing the segmentation processing in each of the at leasttwo first images according to a processing capacity of a system.
 8. Theimage processing device according to claim 7, wherein the processingcapacity of the system is a maximum computing capacity of a computingunit of the system.
 9. The image processing device according to claim 1,wherein using the at least two first images to determine the first depthmap corresponding to the at least two first images under the limit ofthe first disparity threshold further includes: determining ato-be-processed region on each of the at least two first images,respectively; and using to-be-processed regions of the at least twofirst images to determine the first depth map under the limit of thefirst disparity threshold.
 10. The image processing device according toclaim 9, wherein determining a to-be-processed region in each of the atleast two first images further includes: determining a to-be-processedregion according to a processing capacity of a system.
 11. The imageprocessing device according to claim 9, wherein determining ato-be-processed region in each of the at least two first imagesrespectively further includes: estimating an expected moving position ofa movable object; and determining a to-be-processed region in each ofthe at least two first images according to the expected moving positionof the movable object.
 12. The image processing device according toclaim 11, wherein determining a to-be-processed region in each of the atleast two first images according to the expected moving position of themovable object further includes: taking the expected moving position asa center and determining a region matching the expected moving positionon a first image according to a specified region size; and when theregion matching the expected moving position exceeds the first image,modifying the region matching the expected moving position to obtain ato-be-processed region having the specified region size on the firstimage region.
 13. The image processing device according to claim 11,wherein determining a to-be-processed region in each of the at least twofirst images according to the expected moving position of the movableobject further includes: taking the expected moving position as a centerand determining a region matching the expected moving position on afirst image according to a specified region size; and when the regionmatching the movable position exceeds the first image, determining aregion, within the region matching the expected moving position, thatdoes not exceed the first image as a to-be-processed region.
 14. Theimage processing device according to claim 12, wherein, before takingthe expected moving position as the center and determining the regionmatching the expected moving position on a first image according to thespecified region size, the method further includes: determining thespecified region size according to a processing capacity of a system.15. The image processing device according to claim 11, wherein: the atleast two first images are captured by a photographing device on themovable object; and estimating the expected moving position of themovable object further includes: acquiring a current speed of areference object in a photographing device coordinate system; andestimating the expected moving position according to the current speedof the reference object in the photographing device coordinate system.16. The image processing device according to claim 15, wherein acquiringthe current speed of the reference object in the photographing devicecoordinate system further includes: using a current moving speed of themovable object to estimate the current speed of the reference object inthe photographing device coordinate system.
 17. The image processingdevice according to claim 15, wherein acquiring the current speed of thereference object in the photographing device coordinate system furtherincludes: using previously moved positions of the movable object toestimate the current speed of the reference object in the photographingdevice coordinate system.
 18. The image processing device according toclaim 1, wherein the method further includes: acquiring at least twothird images, the third images having the second resolution; using theat least two third images to determine a third depth map correspondingto the at least third images under the limit of the second disparitythreshold; and avoiding obstacles by using the third depth map and thecombined depth map.
 19. The image processing device according to claim1, wherein the method further includes: selecting an image group from aplurality of image groups according to a moving direction of a movableobject, wherein the selected image group includes the at least two firstimages.
 20. The image processing device according to claim 1, whereinthe method further includes: predicting, based on a flight speed of amovable object carrying a photographing device that captures the atleast two first image, a flight trajectory of the movable object; andselecting an image group from a plurality of image groups according tothe flight trajectory of the movable object, wherein the selected imagegroup includes the at least two first images.